Doing MRP on the Server

Moving Files to the Server

The first thing we need to do is get scripts and files onto the server. You can do this with sftp. You can view a tutorial of using sftp here. I’ll show you how it works on my AWS server.

1. Open your Windows terminal.
2. Make sure you’re in the directory that contains your files. You can do this with the cd command or you can put the files you want to move in your home directory - this is where the terminal will be by default.
3. Type sftp root@<url> where <url> is the url of your Digital Ocean droplet. In my instance, I needed to pass the private key, but you don’t have to do that, you’ll have to give a passphrase to login. You should see something like this:

Notice how the command prompt starts with sftp>. This isn’t something you type, it’s just the terminal’s way of telling you that you’re connected using sftp. Next, you’ll need to put files on the server. To see what’s in your local directory, you can type lls and to see what files are currently on the server, you can type ls.

As you can see from the output above, there are a bunch of files locally and none currently on the server. To put the files on the server, say sacl_mrp_script_v2.R, we would use put.

put sacl_mrp_script_v2.R

You will need to do this with all files that get called by your script. If all the files you need get read in from external URLs, then this is the only file you need to upload. Once the files have been successfully transferred, you can type exit to leave the sftp session. The command prompt should change back to the Windows prompt.

Running the Models

Next, you can ssh into the server. You should be able to do this with:

ssh root@<url> 

where <url> is the url of your Digital Ocean droplet. You should see something like this:

The command prompt name should be the same as name of the droplet. Next, you need to make sure that all the packages you’re using are installed in R. You can do this by typing R at the command prompt and you will see the R console.

There, you can install any packages using install.packages('pkgname') just like you would in any R session. Once you’ve done that, you can quit the R window by tying quit('no'). The next step is to run the R script. The trick here is that you will want to do it in such a way that the server keeps executing the script when you get disconnected or log out. If you just run it at the terminal, this won’t be the case. The trick here is to use screen. There is a little tutorial on how it works here. The basic idea is that you start a screen session, run your script, then detach from the screen session. The script will keep running in the background. When you log back in, you can re-attach to the screen session and see the output of your script. At the command prompt, you’ll type screen and hit enter. This will send you to a new screen giving some information about the authors. You will need to hit enter again to get past this screen.

Once you’ve done this, you will see the command prompt again. You can now run your script with:

R CMD BATCH sacl_mrp_script_v2.R

All output will be sent to a file called sacl_mrp_script_v2.Rout. If your file name doesn’t end with .R, for example if your file was sacl_mrp_script_v2.txt, then the output would be the full file name (with extension) followed by .Rout, like sacl_mrp_script_v2.txt.Rout. I’m going to show you how this works using a simple script that just prints a new number every 5 seconds. It’s called long_script.R and its output will be sent to long_script.Rout. In the gif below, you’ll see me get into a screen, set the script running and then detach from the screen. Detaching is done by holding down the ctrl key and then typing a followed by d. You should see a message that you’ve detached from the screen. Then, I’ll use cat to show the R output.

When the screen is detached, you can type cat sacl_mrp_script_v2.txt.Rout any time you want to see the progress. You can always reattach the screen with screen -r. The gif below shows me doing that. Notice that when I do that, the cursor is blinking, but there is no command prompt there. If that’s the case, the script is still running. If you want to stop if from running, you can type ctrl + c, but that will stop the script and you will have to start it again from scratch.

If you want to know if the script is still running, you can type top at the command prompt. That will show you the processes that are running. If you rscript is still running, you will see R show up using up resources, like it is at the top of the output below.

You can exit from the top output by typing ctrl + c.

If your script finishes running, when you reattach the screen, you should see the cursor at a command prompt. When you do ls, you should see the files that you had written out when the script completed.

Generating MRP Estimates

The process of running the MRP estimates is exactly the same as above, but with a different script. You will need to upload the script and any files it calls using sftp. Then, you can ssh into the server, start a screen session, run the script with R CMD BATCH scriptname.R and then detach from the screen. You can check on progress by reattaching to the screen or by looking at the output file with cat scriptname.Rout. When the script is done, you can reattach to the screen and you should see a command prompt. You can then use ls to see if your output files are there.

Some things to think of.

1. Make sure the model objects (the .rds files) are on the server and that they’re being called by the appropriate filenames in the script.
2. Make sure that you’re writing out constituency estimates with distinct filenames, for example with a different suffix than previous ones.

After the estimates are done, you can go to the server and open R. You can read in the constituency estimates and do any aggregation or plotting that you want. You can write out any files you want to your home directory. You can then use sftp to get those files back to your local machine using the get command that works just like put.